Query Processing in Spatial-Query-by-Sketch

Spatial-Query-by-Sketch is the design of a query language for geographic information systems. It allows a user to formulate a spatial query by drawing the desired configuration with a pen on a touch-sensitive computer screen and translates this sketch into a symbolic representation that can the processed against a geographic database. Since the configurations queried usually do not match exactly the sketch, it is necessary to relax the spatial constraints drawn. This paper describes the representation of a sketch and outlines the design of the constraint relaxation methods used during query processing. 1 . Introduction Traditional methods for spatial querying are tedious [17]. The difficulties of communicating a user’s request to a spatial database through conventional spatial query languages becomes most apparent when several users have to work together. Fundamental to this problem is the fact that verbal descriptions of spatial situations are frequently ambiguous and may easily lead to misinterpretations, particularly in multi-language groups. The use of traditional spatial query languages has serious limitations, because geographic concepts are often vague, imprecise, little understood, or not standardized. As an example, take the notion of the spatial predicate cross whose semantics may vary depending on the context in which it is used, the meaning of the objects to which the predicate relates, and the topology and the metric of the particular configuration [39]. These drawbacks make most current spatial query languages error-prone and difficult to use. Graphical user interfaces provide only little improvement for such query languages, because they use the same type of syntax and grammar as the typed languages, and their primary advantage is that they release users from remembering the particular syntax. We attempt to overcome the limitations of conventional spatial query languages by considering alternative interaction methods between users and geographic data in a geographic information system. With the advent of pen-based user interfaces, a more intuitive style of interaction with spatial data is made possible than typing a query or composing it from menus. Pen-based user interfaces are expected to become more important in the future with an increasing demand for multi-media systems in most any application domain [31]. Particularly for the interaction with geographic data, pen-based user interfaces provide a series of advantages over current interaction * This work was partially supported by the National Science Foundation under grant numbers SBR-8810917 (for the National Center for Geographic Information and Analysis), IRI-9309230, IRI-9613646, and SBR-9600465; by Rome Laboratory under grant number F30602-95-1-0042; and by a Massive Digital Data Systems contract sponsored by the Advanced Research and Development Committee of the Community Management Staff and administered by the Office of Research and Development. Journal of Visual Languages and Computing, Vol. 8, No. 4, pp. 403-424, 1997. techniques. By their very nature, geographic data are spatial and it is most appealing to refer to them in terms of explicit spatial concepts. Rather than expressing a spatial query in lexical terms, users may prefer to sketch a spatial query. Sketching is an interaction mode that more directly supports human spatial thinking than interactions through a verbal spatial query language alone, because users frequently have an image-like representation in their minds when they query about spatial configurations. It also provides immediate graphical feedback and, therefore, is an inherently more natural process to formulate many spatial constraints than a textual language. In lieu of forcing users to express a spatial configuration in some (semi)-formal or natural language, it is a major step towards the successful use of spatial information systems if users are allowed to draw a picture of the image they have in their minds, in order to retrieve the spatial data of interest. Such a spatial query language is Spatial-Query-by-Sketch [10], which will allow users to express spatial queries closer to the way they think about many spatial problems and incorporates powerful reasoning mechanisms to infer geometric variations in the sketch. Spatial-Query-by-Sketch is a design and a prototype implementation is currently under development. An area of particular interest is the access to digital image libraries [22, 42] through a language like Spatial-Query-bySketch, where users may want to retrieve, for instance, remotely-sensed images on which features match a particular geometric configuration drawn. Besides many interesting considerations about interaction by sketching, Spatial-Query-bySketch poses challenging questions with respect to the processing of sketched queries. If the database were to retrieve only those configurations that provide an exact match with the geometry of the drawing, standard methods used in image matching and image retrieval could be applied. In a geographic context, however, it may be necessary to relax some of the constraints of the sketch, because trying to retrieve a situation that fits exactly the geometry of the sketch would only rarely result in a match. There is an important conceptual difference, however, between finding a picture that matches a sketch vs. finding a geographic configuration that matches a sketched query. In pictorial queries, the shape of the objects, their relative sizes, and their proportions are considered to be known [26]. The match between the picture and the sketch—the outline of some features that must appear on the image of interest—could be established through modest variations of the metric. The processing task is then to match the outlines with the boundaries on the pictures. Deviations between the image and the sketch occur due to inevitable inaccuracies in the user’s drawing. To compensate for them, methods like epsilon bands around the boundaries, within which valid matches would be found, are acceptable solutions. In queries about geographic data, however, this is not the case, because such spatial properties as the orientation of the objects may be immaterial for the query or relative distances among the objects may be highly distorted. To decide which constraints might be relaxed and which constraints should be maintained, it is necessary to base the query processing on a computational model for similarity of spatial relations. For this goal, we use a powerful computational model to represent spatial relations and extend this model where necessary to account for various degrees of similarity. This approach enables us to retrieve not only those situations that provide a perfect match with the sketch, but also those that capture the essence of the sketch; therefore, Spatial-Query-by-Sketch enables spatial similarity retrieval [26]. Experiments in psychology and cartography showed that topology is among the most critical information people refer to when they assess spatial relationships in geographic space [38, 49, 52], while metrical changes are frequently considered to be of lesser importance. To reflect such human behavior, Spatial-Query-by-Sketch is based on the premise topology matters, metrical refines [19]. The remainder of this paper first reviews previous approaches to spatial querying, focusing on traditional spatial query languages, visual spatial query languages, and sketching. Section 3 introduces the principles of Spatial-Query-by-Sketch and gives a guided tour through some fundamental interactions in Spatial-Query-by-Sketch. Section 4 focuses on the internal representation of a sketched query in the form of a semantic network with spatial objects and their spatial relationships. Query processing of spatial relations, relaxation of spatial constraints, Journal of Visual Languages and Computing, Vol. 8, No. 4, pp. 403-424, 1997. prioritization of query results are described in Sections 5, 6, and 7, respectively. Conclusions and future work are discussed in section 8. 2 . Spatial Querying Spatial-Query-by-Sketch builds on state-of-the-art knowledge in spatial query languages, particularly visual spatial query languages, and extends the sketching paradigm. This section reviews relevant approaches in these fields. 2 .1 . Spatial Query Languages Query languages for geographic databases and geographic information systems are either complex macro languages or extensions of SQL. There exists a large variety of Spatial SQL dialects [7, 28, 32, 47]. Such SQL extensions are relevant to Spatial-Query-by-Sketch, because they provide the means for accessing geographic databases and retrieving data from a database. Most critical is the support for spatial relations. Many SQL dialects include some notions of spatial relations, however, the semantics of the operations provide varying levels of detail and differ quite dramatically. Spatial extensions to SQL are currently being addressed by the SQL3 Multimedia working group. Similar to SQL extensions, there are several spatial query languages that are derivatives of Query-by-Example. Query-by-Pictorial-Example [5] and Picquery [33] are examples for the Query-by-Example approach of inserting example values in tables, without exploiting the 2dimensional characteristics of the language for spatial (2-dimensional) querying. 2 .2 . Visual Spatial Query Languages More advanced user interfaces and spatial query languages include concepts similar to SpatialQuery-by-Sketch. The query language Cigales, for example, allows users to draw a query [4]. Unlike Spatial-Query-by-Sketch, Cigales requires the users, prior to drawing the sketch, to select the type of spatial relation they are addressing [37]. For instance, to specify that the road enters the park, the user would have to select the “intersect” operation, and then draw the particular configuration [1]. This leads to moded interfaces, which are tedious to use. In a similar attempt, Lee and Chin [36] designed an iconic query language in which users compose a query by selecting spatial relations from a predefined set represented as icons. They only consider a small subset of topological relations, so that a user can select them from a set of icons. A visual spatial query language that is based on a comprehensive algebra is Query-by-VisualExample [43], an extension of Query-by-Example. Users of Query-by-Visual-Example construct templates of scenes in an array-like framework, describing primarily cardinal directions. While this approach comes closer to the way people think about space and its objects, it has its limitations through the equal resolution of the space. The grid also favors the specification of direction relations, but makes it more difficult to state approximate distances and topological relations independent of directions. All of these visual spatial query languages lack a method to cope with the fact that an acceptable answer—even the best fit—may actually differ from the geometry in the query configuration. 2 .3 . Sketching Sketching was used in the past primarily in CAD for design. Sketchpad [51] and ThingLab [2] were initial approaches to formulate constraints graphically. Pizano et al. [46] used spatial constraints for describing consistency in spatial databases; however, unlike describing situations that should match the configuration of interest, they focused on constructing those situations that would establish unacceptable database states. Although their language was iconic rather than sketch-based, it shares much similarity with the principles of sketching. Sketching for querying was used in Query by Visual Example [29, 30, 34, 35] and Query by Image Content [22], which are targeted for content-based image retrieval. While the interaction Journal of Visual Languages and Computing, Vol. 8, No. 4, pp. 403-424, 1997. mode of these query languages is similar to the basics of Spatial-Query-by-Sketch [10]—in both cases users draw an approximate spatial configuration of what to retrieve—scope and sketch interpretation are considerably different. Sketches for content-based image retrieval assume that the user draws something that matches quite closely the target and that all relations are intended as drawn. Their query processors accommodate primarily metrical variations and they are very sensitive to variations in sizes, orientations, and shapes. On the other hand, Spatial-Query-bySketch assumes that the user’s sketch and the targets may vary considerably, as long as they match in the most important criteria. Spatial relations have been considered as a secondary criterion in an image retrieval system that focuses on shape similarity [6]. The measures for shape are quantitative and thus expensive to process in a spatial database, and the spatial relations considered use rough approximations based on minimum-bounding rectangles. In contrast, Spatial-Query-by-Sketch prefers qualitative measures, starting with the spatial relations among the objects drawn, and resorts to quantitative methods only to prioritize hits. The concepts of Spatial-Query-by-Sketch come closest to the Electronic Cocktail Napkin [25], which uses free-hand drawings to interact with architectural images, and a query language for sketch-based querying of geographic databases [40]. While their interactions modes and intention for similarity retrieval closely match with Spatial-Query-by-Sketch, their models used for representing sketches and processing them use an ad-hoc collection of spatial relations, which distinguishes Spatial-Query-by-Sketch as it is founded on a solid mathematical model of spatial relations and their relaxations. 3 . Spatial-Query-by-Sketch Spatial-Query-by-Sketch is designed to use a touch-sensitive input device—ideally a touch screen with a pen, such as Apple’s Newton. Simulations may be obtained with a mouse or a trackball, but sketching with these devices is more cumbersome and therefore less effective. Users draw with a pen a geometric configuration that matches closely the spatial situation(s) they expect to retrieve from the geographic database. While composing the sketch, they may annotate the sketch to describe desired properties of the sketched objects. Spatial-Query-by-Sketch parses the sketch and translates it into a topological vector data model [27]. Subsequently, Spatial-Query-by-Sketch develops a query processing plan and executes the query against the spatial database. If several scenes match the query, the results are prioritized such that scenes with the best match to the query are presented first. The following scenario provides a cursory outline of the envisioned interaction a user may perform when sketching a query. This user interface is organized into three major interaction areas: the sketch region in which the user draws the configuration of interest; the overview area which displays the sketch in its entirety and allows users to pan and zoom; and the control panel from which the user selects database commands, the type of feature he or she is drawing, and the confidence level for the placement of a feature. Users employ a pen to sketch an example of what they want to find in the database. In this particular case, the user is interested in all land parcels that have a wooded area and a river crossing the parcel. The user first sketches the parcel by selecting the class of the object (in this case a Parcel), and drawing its boundary (Figure 1a). Then she describes the location of the forest by drawing part of the forest’s boundary (Figure 1b). Since it is unclear on which side of the line the forest is located, the user fills the interior of the forest (Figure 1c). Finally the user draws a river such that it crosses the land parcel (Figure 1d). Since the user is satisfied with the drawing, she requests that all configurations that match the sketch be retrieved from the database by pressing the Go! button on the control panel. Journal of Visual Languages and Computing, Vol. 8, No. 4, pp. 403-424, 1997. Figure 1: (a) The user draws the geometry of a land parcel; (b) the user adds the boundary of a forest; (c) to determine the location of the forest, the user fills the forest’s interior; and (d) the user adds the location of a stream such that it crosses the land parcel, but does not intersect with the forest. 4 . Symbolic Representation of a Sketch While a bitmap representation would provide an accurate snapshot of such a sketch, it would be difficult to interpret it and match it against elements in other datasets whose relations, sizes, and shapes are distorted or not to scale with the sketch or whose orientations among elements differ to some degree. Instead, we select an object representation for the sketch, which allows us to abstract away some details of the sketch while it emphasizes its salient parts. This representation stresses objects, their spatial and non-spatial properties, and the spatial relations among the objects drawn. The latter are of particular importance for processing a query in Spatial-Query-by-Sketch as they capture the essence of a spatial scene. We represent the sketch internally as a semantic network of spatial objects and their binary spatial relations. In this network, each object drawn corresponds to a node whose values are given by the semantics assigned in the sketch. They may include the class of an object, a name, other attribute values, or such metrical constraints as the size of the area or length of an object. Directed Journal of Visual Languages and Computing, Vol. 8, No. 4, pp. 403-424, 1997. edges between nodes stand for binary spatial relations between the spatial objects. For this purpose, we distinguish five different types of spatial relations: coarse binary topological relations, detailed binary topological relations, metrical refinements, coarse cardinal directions, and detailed cardinal directions. With these five types of binary spatial relations, a qualitative model of a sketch is built in the form of a multi-resolution semantic network, called a scene network. Such a network serves as a symbolic, qualitative representation of the sketch. Its elements translate into predicates in spatial queries. See [44] for a discussion about the completeness of the approach of using binary relations for spatial queries. The scene network may be constructed at different levels of detail, for instance only at a coarse level of detail with topological and direction relations, or only as a topological representation with coarse and detailed topological relations. For the most detailed analysis, a complete scene network would be derived with all five types of spatial relations. Such a representation translates easily into database queries in the form of first-order predicates or extended-SQL statements. Depending on the configuration, fewer binary relations may be sufficient to describe the scene completely if they allow to drive uniquely the eliminated relations through compositions of elementary or inferred relations [20]. There are additional dependencies among the different types of binary relations that could further reduce the smallest number of relations required to fully specify a scene. For example, detailed cardinal directions imply their corresponding coarse cardinal directions. The actual number of spatial relations to be considered for processing a particular query is an issue of spatial query optimization [8]. In the following, we discuss the models used for the five types of spatial relations. 4.1 Coarse Topological Relations We base the analysis of topological relations on the 9-intersection, a comprehensive model for binary topological relations that applies to objects of type area, line, and point [13, 15]. It characterizes the topological relation between two point sets, A and B , by the set intersections of A ’s interior, boundary, and exterior with the interior, boundary, and exterior of B , called the 9intersection. With each of these nine intersections being empty or non-empty, the model has 512 possible topological relations between two point sets, some of which cannot be realized. For two simple regions without holes embedded in R2, the categorization shows eight distinct topological relations. They have been called disjoint, meet, equal, overlap, inside, contains, covers, and coveredBy (Figure 2). For two simple lines (non-branching, no self-intersections) embedded in R2, 33 different topological relations can be realized with the 9-intersection, and for a line and a region, 19 different situations are found [16].